Voice control user interface during low-power mode

ABSTRACT

A method include placing a first processor in a sleep operating mode and running a second processor that is operative to wake the first processor from the sleep operating mode in response to a speech command phrase. The method includes identifying, by the second processor, a speech command phrase segment and performing a control operation in response to detecting the segment in detected speech. The control operation is performed while the first processor is maintained in the sleep operating mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/282,303, filed Sep. 30, 2016, which is a continuation of Ser. No.15/281,843, filed Sep. 30, 2016, which is a continuation of U.S.application Ser. No. 14/197,633, filed Mar. 5, 2014, which claims thebenefit of U.S. Provisional Application No. 61/922,621, filed Dec. 31,2013.

U.S. application Ser. No. 14/197,633 is a continuation-in-part of U.S.application Ser. No. 13/830,452, filed Mar. 14, 2013, which claims thebenefit of U.S. Provisional Application No. 61/757,690 filed Jan. 28,2013; U.S. Provisional Application No. 61/736,536 filed Dec. 12, 2012;and U.S. Provisional Application No. 61/720,322 filed Oct. 30, 2012.

U.S. application Ser. No. 14/197,633 is a continuation-in-part of U.S.application. Ser. No. 14/142,177, filed Dec. 27, 2013, which claims thebenefit of U.S. Provisional Application No. 61/860,725 filed Jul. 31,2013; U.S. Provisional Application No. 61/827,723 filed May 27, 2013;U.S. Provisional Application No. 61/798,097 filed Mar. 15, 2013; andU.S. Provisional Application No. 61/776,793 filed Mar. 12, 2013.

U.S. application Ser. No. 14/197,633 is a continuation-in-part of U.S.application Ser. No. 14/142,210, filed Dec. 27, 2013, which claims thebenefit of U.S. Provisional Application No. 61/892,527 filed Oct. 18,2013; and U.S. Provisional Application No. 61/857,696 filed Jul. 23,2013.

U.S. application Ser. No. 14/197,633 claims the benefit of U.S.Provisional Application No. 61/878,295, filed Sep. 16, 2013.

U.S. application Ser. No. 14/197,633 claims the benefit of U.S.Provisional Application No. 61/860,725, filed Jul. 31, 2013.

The contents of each of these applications are hereby incorporated byreference in their entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to voice recognition systemsand more particularly to user interfaces that include voice recognition.

BACKGROUND

Many existing communication devices (also referred to as mobile devices)include a means of notifying a user that an event occurred with respectto the mobile device. Such an event may be a missed phone call, incomingemail message or incoming text message, etc. One example means that hasbeen used to inform a user of such notifications is a light emittingdiode (LED). The LED may blink occasionally to alert the user to statechanges of the mobile device, such as when an email or text message hasbeen received, or when a telephone call has been missed, where the statechange results in a notification. In some mobile devices, the LED mayblink even when the mobile device display is in a power save mode orwhen the mobile device is in a sleep mode of operation.

When the mobile device is awake, and therefore not in a sleep mode ofoperation, the user may observe such notifications directly on themobile device display by way of visual indications such as textualnotifications, icons or by other graphic images that provide additionalinformation regarding the specific type of notification. In anotherexample of visual indications, most mobile devices will display at leasta clock showing the time of day and a battery charge level. If themobile device is in a sleep mode and the display is turned off, the usermust wake the mobile device in order to turn the display on again toview the visual indications such as the time of day, or to accessinformation about specific notifications. For control systems employingvoice control, viewing visual indications related to notifications mayrequire waking the mobile device, more particularly waking a mainprocessor, in order to implement desired functionality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a mobile device in accordance with theembodiments.

FIG. 2 is a block diagram of another mobile device in accordance withthe embodiments.

FIG. 3 is a flowchart of an example method for generating a speech modelin accordance with an embodiment.

FIG. 4 is a flowchart of an example method for performing a low powermode operation in accordance with an embodiment.

FIG. 5 is a flowchart of an example method for revealing helpinformation to a user in accordance with an embodiment.

FIG. 6 is a flowchart of an example method for performing a forcedbreath operation on a display in accordance with an embodiment.

FIG. 7 is a flowchart of an example method for displaying information toa user in accordance with an embodiment.

FIG. 8 is a flowchart of an example method for prompting a user for avoice command in accordance with an embodiment.

FIGS. 9A through 9D provide examples of display data displayed on amobile device display in accordance with various embodiments.

FIG. 10 is a block diagram of an example control system in accordancewith an embodiment.

DETAILED DESCRIPTION

Briefly, the present disclosure provides apparatuses and methods for avoice control user interface with progressive command engagement. Inaccordance with the embodiments, a voice command phrase may be brokendown or decomposed into speech command phrase segments which represent aportion of the voice command phrase. The speech command phrase segmentsare then utilized, in accordance with the embodiments, to enable a firsttype of user interaction with an apparatus that may, for example, occurduring a low-power state. The user may speak additional speech commandphrase segments to engage in other user interactions of the first type,or may speak the entire voice command phrase to engage in a userinteraction of a second type. That is, speaking speech command phrasesegments initially and then subsequently speaking an entire commandphrase progressively activates additional capabilities of the apparatusprogressing from user interactions of the first type to userinteractions of the second type. For example, a speech command phrasesegment may allow the user to access features during a low-power state(i.e. user interactions of a first type), while the full voice commandphrase may access features that require waking an application processor(i.e. user interactions of a second type). The voice control userinterface thereby provides the user with progressive command engagement.

The present disclosure provides, among other things, a method ofoperation that begins with operating a first processor in a sleep modeand operating a second processor that is operative to wake the firstprocessor from the sleep mode in response to a speech command phrase.The method includes identifying, by a speech recognition engine, aspeech command phrase segment and performing, by the second processor, acontrol operation in response to the segment. Performing a controloperation by the second processor in response to the segment may includeperforming the control operation without waking the first processor fromthe sleep mode. Also, performing a control operation by the secondprocessor in response to the segment may include performing the controloperation prior to waking the first processor from the sleep mode, andalso waking the first processor from the sleep mode. Performing thecontrol operation without waking the first processor from the sleep modemay include providing output information in response to the segment.

In some embodiments, providing output information in response to thesegment may include updating a portion of display data maintained in aframe buffer, where the display data corresponds to a most recentlydisplayed image, and displaying the updated display data on a display.In such embodiments, the method may further include activating at leasta portion of the display prior to displaying the updated display data,where the activated portion of the display corresponds to a portion ofthe most recently displayed image that is updated according to theupdated display data.

In some embodiments, displaying the updated display data on a displaymay include displaying the updated display data on the display for apredetermined period of time, and turning the display off after thepredetermined period of time has expired. Also in some embodiments,providing output information in response to the segment may includeproducing an audible response or a mechanical movement as the outputinformation.

The present disclosure also provides an apparatus operative to performvarious methods of operation disclosed herein. One example apparatus mayinclude a first processor having a sleep operating mode, a speechrecognition engine operative to identify a speech command phrase segmentwhile the first processor is operated in the sleep operating mode, and acontrol system operatively coupled to the first processor. The controlsystem is operative to control equipment in response to the speechcommand phrase segment, while the first processor operates in the sleepoperating mode. A second processor is operatively coupled to the firstprocessor, to the speech recognition engine and to the control system.The second processor is operative to wake the first processor from thesleep operating mode in response to a speech command phrase, and to sendcontrol command data to the control system also in response to thespeech command phrase segment.

In some embodiments, the apparatus may also include a basic speech modeland a segment speech model. The speech recognition engine is operativeto utilize the segment speech model to identify the speech commandphrase segment when the first processor operates in the sleep operatingmode. In some embodiments, the second processor is also operative tosend control command data to the control system, in response to thespeech command phrase segment, without waking the first processor fromthe sleep mode.

In another embodiment, an apparatus includes a first processor having asleep operating mode, a speech recognition engine operative to identifya speech command phrase segment while the first processor is operated inthe sleep operating mode, display hardware and a display. The displayhardware is operatively coupled to the first processor and is operativeto store display data in a frame buffer for a most recently displayedimage while the display is turned off. A second processor is operativelycoupled to the first processor, to the speech recognition engine, and tothe display hardware. The second processor is operative to wake thefirst processor from the sleep operating mode in response to a speechcommand phrase, and to update a portion of the display data in the framebuffer in response to the speech command phrase segment. The secondprocessor is also operative to turn at least a portion of the display onfor a period of time to display the updated portion of the display data,and to turn the display off again after expiration of the period oftime.

The apparatus of may also include a basic speech model and a segmentspeech model. The speech recognition engine is operative to utilize thesegment speech model to identify the speech command phrase segment whenthe first processor operates in the sleep operating mode. The secondprocessor and the display hardware are also operative to activate atleast a portion of the display prior to displaying the updated displaydata. More particularly, the activated portion of the displaycorresponds to a portion of the most recently display image that isupdated according to the updated display data.

Turning now to the drawings, a block diagram of an example apparatus ofan embodiment, a mobile device 100, is provided in FIG. 1. One or moreinternal connection buses 105 provide operative coupling between anapplication processor 101, low-power processing 103 and other variousmobile device 100 components. As used herein, components may be“operatively coupled” when information can be sent between suchcomponents, even though there may be one or more intermediate orintervening components between, or along the connection path. Therefore,any of the various components connected to the one or more connectionbuses 105 may be understood herein to be operatively coupled to theapplication processor 101, to the low-power processing 103, or someother component, etc. where appropriate.

The memory 117 is a non-volatile, non-transitory memory, and storesexecutable instructions (executable code) or data that may be accessedand executed by the application processor 101 or the low-powerprocessing 103. The mobile device 100 includes one or more microphones121 (such as a microphone array) and one or more speakers 123 that areoperatively coupled by connections 125 to audio configuration andpre-processing logic 119. The audio configuration and pre-processinglogic 119 may include analog-to-digital converters (ADCs),digital-to-analog converters (DACs), echo cancellation, high-passfilters, low-pass filters, band-pass filters, adjustable band filters,noise reduction filtering, automatic gain control (AGC) and other audioprocessing that may be applied to filter noise from audio received usingthe one or more microphones 121. The audio configuration andpre-processing logic 119 may be a single component as shown in FIG. 1 ormay be implemented partly in hardware and partly in software or firmwareexecuted by application processor 101 or by low-power processing 103. Insome embodiments, the audio configuration and pre-processing logic 119may be implemented using several hardware components and may alsoutilize one or more software or firmware components in variouscombinations. The audio configuration and pre-processing logic 119 maybe operative to control the one or more microphones 121 to turn certainmicrophones on or off or to adjust filtering or gain for purposes ofvarious applications.

A first wireless transceiver 113 may provide wireless communicationcapabilities for one or more wide area network communications systemssuch as, but not limited to, cellular, 2G, 3G or 4G wirelesscommunications systems and may also provide Internet connectivity over awireless interface. In some embodiments, a second wireless transceiver115 may also be present in the mobile device 100. The second wirelesstransceiver 115 may provide wireless connectivity capabilities such as,but not limited to, Wi-Fi™, Bluetooth™, Wireless USB™, ZigBee™, or othertechnologies, etc., and may also provide Internet connectivity over anyof these wireless interfaces.

The user interface 111 may include voice control, a track ball mouse,touch sensitive elements, physical switches, gyroscopic positionsensors, etc. The user interface 111 may provide command and controlsignals to the application processor 101 or to the low-power processing103 during various modes of operation of the mobile device 100. Adisplay 109 may provide a touchscreen capability as part of the userinterface 111, and may also provide a graphical user interface (GUI)related to a mobile device operating system, a GUI related to one ormore mobile device applications, etc. The display 109 is coupled todisplay hardware 107 by display bus 108 and the display hardware 107 isoperatively coupled to the application processor 101 and to thelow-power processing 103 by the one or more connection buses 105. Thedisplay hardware 107 is part of a control system that controls thedisplay 109.

FIG. 2 is a block diagram of another example apparatus, in anotherembodiment, which is a mobile device 200. Mobile device 200 includessimilar components to the components of mobile device 100, but theseother components are omitted from FIG. 2 for clarity. One or moreconnection buses 105 provide operative coupling between an applicationprocessor 101, a low-power processing 103, non-volatile, non-transitorymemory 117, and display hardware 107. The display hardware 107 isoperatively coupled to a display 109 by a display bus 108. The memory117 stores executable instructions (also referred to herein asexecutable code) and data for one or more operating systems 201 and forvarious applications 203. The memory 117 may also store voicerecognition engine 205 executable code and speech segment monitor 206executable code in accordance with an embodiment. The memory 117 alsostores a basic speech model 209, a trained speech model 211 and asegment speech model 213 in some embodiments. These speech models areused by the voice recognition engine 205 and the speech segment monitor207. The speech segment monitor 207 is itself a voice recognitionengine, but may be a reduced version of the voice recognition engine 205that is suitable for low-power mode operation. In other words, thespeech segment monitor 207 does not implement all features of the fullvoice recognition system implemented by the voice recognition engine205. The voice recognition engine 205 may be implemented by theapplication processor 101 executing the executable code stored inmemory, or may be implemented using hardwired circuitry such as, but notlimited to, an ASIC (application specific integrated circuit) orequivalent.

The memory 117 may also store display manager 214 executable code anddisplay graphics 223. Thus in some embodiments, a display manager may beimplemented by a processor executing the display manager 214 executablecode. However, in other embodiments, a display manager 215 isimplemented using hardwired circuitry such as an ASIC or equivalent. Inother embodiments, a display manager may be implemented as a combinationof hardware and software. The application processor 101, as well as someprocessors included in the low-power processing 103, are thereforeoperative to access the memory 117 to access data or to execute storedexecutable instructions. The low-power processing 103 is thereforeoperative to access data and to execute some of the executableinstructions stored in memory 117. For example, the low-power processing103, in accordance with one example embodiment is operative to executethe speech segment monitor 206 executable code and the display manager214 executable code. In another example embodiment, the speech segmentmonitor 207 is an ASIC, and the display manager 215 is an ASIC, andneither ASIC requires any additional executable code in order tooperate. Regardless of the specific implementation utilized in thevarious embodiments, the speech segment monitor 207 may access andutilize the basic speech model 209, the trained speech model 211, andthe segment speech model 213. The low-power processing 103 may accessdisplay graphics 223 and provide portions of the display graphics 223data to the display hardware 107 over the one or more connection buses105. In one example embodiment, the display hardware 107 includes aframe buffer 217 and display drivers 219 that store and refresh displaydata 221 displayed by the display 109.

The memory 117 may also store various libraries (not shown) used toimplement various application programming interfaces (APIs). For examplean API may enable the voice recognition engine 205 to send voicecommands to the various applications 203 or to the one or more operatingsystems 201. The various applications 203 may include, but are notlimited to, web browsers, email programs, instant messaging clients,etc., any of which may communicate with the Internet or other networks.In another example, an API may enable the speech segment monitor 207 tosend command and control data to the display manager 215 or to the voicerecognition engine 205.

It is to be understood that any of the above described softwarecomponents (i.e. executable instructions or executable code) or any ofthe above described components of mobile device 200 may be implementedas software or firmware (or a combination of software and firmware)executing on one or more processors, or using ASICs(application-specific-integrated-circuits), DSPs (digital signalprocessors), hardwired circuitry (logic circuitry), state machines,FPGAs (field programmable gate arrays) or combinations thereof.Therefore the mobile device 200 illustrated in FIG. 2 is one example ofa mobile device and is not to be construed as a limitation on thevarious other possible mobile device implementations that may be used inaccordance with the various embodiments. The low-power processing 103may be a single component or may be implemented as any combination ofDSPs, ASICs, FPGAs, CPUs running executable instructions, hardwiredcircuitry, state machines, etc., without limitation. Therefore, asalready described in one example provided above, the speech segmentmonitor 207 may be implemented as one ASIC, and the display manager 215may be implemented as another ASIC that is operatively coupled to thespeech segment monitor 207. The speech segment monitor 207 isoperatively coupled to the application processor 101 by the one of moreconnection buses 105, and is operative to provide a “wake-up” signal tothe application processor 101 in response to detecting a wake-up voicecommand phrase received by the one or more microphones 121. The speechsegment monitor 207 is also operative to provide a control signal to thedisplay manager 215 in response to detecting a speech command phrasesegment. The operation of detecting a speech command phrase segment isdescribed in further detail herein below.

The various embodiments of the present disclosure also includenon-volatile, non-transitory computer readable memory, other than memory117, that may contain executable instructions (i.e. executable code),for execution by at least one processor, that when executed, cause theat least one processor to operate in accordance with the functionalityand methods of operation herein described. The computer readable memorymay be any suitable non-volatile, non-transitory, memory such as, butnot limited to, programmable chips such as EEPROMS, flash ROM (thumbdrives), compact discs (CDs) digital video disks (DVDs), etc., that maybe used to load executable instructions or program code to otherprocessing devices such as servers, mobile devices or other devices suchas those that may benefit from the features of the herein describedembodiments. For example, the speech segment monitor 206 executablecode, display manager 214 executable code or display graphics 223 may bestored on any of the above described forms of computer readable memory,etc.

In accordance with the embodiments, the mobile device 200 is configuredto implement a touchless control feature (i.e. voice control feature)that enables a user to speak a specific trigger phrase, followed by acommand, to wake the mobile device 200 (if not already awake) from asleep mode of operation. In order to conserver battery power, the mobiledevice 200 may be placed in a sleep mode of operation from time-to-timebased on remaining idle for some preset period of time. During sleepmode, the application processor 101 is partially shut down and maintainsonly those functions necessary for basic system operation. In someembodiments, the low-power processing 103 takes over some operations ofthe application processor 101 and may execute at least a reduced codeversion of the voice recognition engine 205 executable code. In otherembodiments, the speech segment monitor 207 takes over when theapplication processor 101 goes into sleep mode, and provides a limitedvoice recognition capability. The speech segment monitor 207 voicerecognition engine listens for voice commands, processes the commands,and returns a user interface action or behavior that is the result ofthe user's intent. For example, if the user wishes to view visualindications on the display the user may speak a previously trained“wake-up” command phrase to wake the application processor 101 and turnthe display 109 on. The low-power processing 103, or more specificallythe speech segment monitor 207, then sends a wake command to theapplication processor 101 over the one or more connection buses 105 toactivate the application processor 101 and turn the display back on,etc.

However, if the user only wants to see, for example, the time of day, itis not a necessity to wake the application processor 101. The mobiledevice 200 is therefore also configured to implement “breathing” of thedisplay 109. The breathing process turns the display 109 onoccasionally, and then off again, using a reduced frequency clock rateduring sleep mode, such that the user may see the display and any newvisual indications that may be present. Turning briefly to FIG. 9A, anexample of a display “breath” is shown in which the display 109 may belit up to show only the background (such as a wallpaper) and some visualindications 901. The visual indications 901 are examples and include aclock (i.e. time of day indication), an envelope icon indicating anemail or text message received, and a battery level indicator. Thebattery level indicator may only be displayed when the battery charge isreduced to a certain predetermined level and therefore may not always bepresent. The envelope icon will only be present if an unread message hasbeen received, etc. Some other icons may also be present such as a“missed call” icon, or an icon for unlocking the mobile device, or someother icon.

Returning to FIG. 2, the low-power processing 103 implements thebreathing process by way of the display manager 215. The last graphicimage displayed prior to sleep mode, as represented by display data 221,is latched in the frame buffer 217 and displayed on display 109 onlyduring a “breath.” The display manager 215 is operative to write to theframe buffer 217 and override portions of the display data 221. In oneexample, the low-power processing 103 may update the clock (time of day)the battery charge level and notification icons and the display manager215 may update the appropriate portions of the frame buffer 217. Theremainder of the display data 221 remains unchanged while theapplication processor 101 is in sleep mode. The low-power processing103, display manager 215, therefore handles the breathing operations andupdating of the frame buffer 217 portions. The display graphics 223 isdisplay data that includes the notification icons and other graphicsobject, such as the envelope shown in FIG. 9A, that may be retrieved bythe display manager 215 and sent to the frame buffer 217 to update thedisplay data 221. Put another way, the display manager 215 uses displaydata stored as display graphics 223, to overwrite portions of thedisplay data 221 that is maintained by the frame buffer 217.

In the embodiments, battery power can be conserved by performing adisplay breath operation only when desired by the user. Therefore, inaccordance with the embodiments, the mobile device 200 is alsoconfigured to perform a “forced breath” display operation. Morespecifically, the various embodiments provide user voice commandengagement in varying degrees (i.e. progressive engagement) by using thesegment speech model 213. Rather than performing breathing and turningthe display on and off on a schedule, a “forced breath” operation turnsthe display on and off again only in response to a user voice command.Both scheduled breathing operations and forced breathing may be usedtogether in the various embodiments. The forced breath may also occur inresponse to other interactions with the user interface 111, such as bynudging the mobile device 200 or by touching the touch sensitive display109. For voice commands, if the display 109 is off and a speech commandphrase segment is spoken, the low-power processing 103 will trigger aforced breath immediately, without waking up the application processor101. For example, the trained speech model 211 may contain a speechcommand phrase such as “Okay, wake up!” If the speech segment monitor207 detects this “wake-up speech command phrase,” then the speechsegment monitor 207 will send a wake-up command to the applicationprocessor 101. However, in accordance with the embodiments, the segmentspeech model 213 may include the word “Okay” as a speech command phrasesegment. In that case, the speech segment monitor 207 will send a forcedbreadth command to the display manager 215 in response to detecting thespeech command phrase segment. In addition, if the user speaks theentire speech command phrase such as “Okay, wake up,” the speech segmentmonitor 207 will also send the wake-up command to the applicationprocessor 101 to begin the wake-up process. In other words, the forcedbreadth may occur prior to the application processor 101 being fullyawakened. However, if the user only speaks the speech command phrasesegment, the speech segment monitor 207 will only send the forcedbreadth command to the display manager 215 and will not wake theapplication processor 101. Therefore, the user is provided with outputinformation, as the display data 221 that is displayed on the display109 for a brief period of time in response to detection of the speechcommand phrase segment.

Therefore, in accordance with the embodiments, the speech segmentmonitor 207 and the display manager 215 operate when the mobile device200 is placed in a low power mode, such as a sleep state, and may wakethe mobile device 200 (i.e. by waking the application processor 101)depending on the voice commands detected by the speech segment monitor207. Example methods of operation that involve the speech segmentmonitor 207, the display manager 215 and the low-power processing 103are described below with respect to flowcharts in FIG. 3 through FIG. 8.

FIG. 3 is a flowchart of an example method for generating a speech modelin accordance with an embodiment. As is known by those familiar withvoice-recognition systems, a user may train a voice recognition systemto recognize the user's particular speech pattern. A voice recognitionengine may be implemented in some embodiments by the applicationprocessor 101 executing voice recognition engine 205 executable code.The voice recognition engine enables the user of the mobile device 200to train the voice recognition engine and to create the trained speechmodel 211 for specific command words or command phrases. As mentionedabove, the voice recognition engine may be implemented by theapplication processor 101 executing the voice recognition engine 205executable code. However, in some embodiments, a voice recognitionengine may be implemented as a separate component such as a DSP, ASIC orsome other processor etc. At the time the user creates the trainedspeech model 211 and sets up touchless control features, the user mayalso access the speech segment monitor 207. In some embodiments, thespeech segment monitor 207 may be integrated with a voice recognitionengine or may be a separate software component or hardware component asshown in FIG. 2. The speech segment monitor 207 may operate on thetrained speech model 211 either as the trained speech model 211 is beinggenerated by the user (such as during training), or subsequently byperforming operations on the completed trained speech model 211 in apost-processing manner. Turning to FIG. 3, in operation block 301, thespeech segment monitor 207 searches for and detects one or more pausesin one or more command phrases contained within the trained speech model211 either during initial creation or during post-processing. Inoperation block 303, the speech segment monitor 207 will select asegment of a trained command phrase based on the location of at leastone pause. The speech segment monitor 207 may detect that the commandphrase contains a series of words as indicated by pauses in the speechpattern. For example, the user may create a command phrase specificallyfor waking up the application processor 101 from sleep mode. A commandphrase such as “Okay, wake-up” may be used as was discussed in aprevious example above. The speech segment monitor 207 may detect apause between the first word “okay” and the second portion of thecommand phrase “wake-up”. In operation block 305, the speech segmentmonitor 207 may store the segment (i.e. the word “okay”) in a segmentspeech model 213. This operation of the speech segment monitor 207 maybe repeated for any number of command phrases in order to create thesegment speech model 213. Subsequently, as shown in operation block 307,the speech segment monitor 207 may use the segment speech model 213 (andthe speech segments contained therein) as commands for implementinglow-power control functions and features such as, but not limited to, aforced breath operation of the display 109. Alternatively, the basicspeech model 209 may contain default command phrases that are used forvarious purposes. Therefore, in another example, the phrase “Okay wakeup” may be a default command phrase for waking the application processor101 (and therefore for waking the mobile device 200). In that case, thesegment speech model 213 will include default speech command phrasesegments that have been derived from the default command phrases of thebasic speech model 209. Put another way, a “speech command phrasesegment” is determined by identifying pauses within a command phrasewhere the pauses may define a word, one or more speech syllables, orother vocalization that may be subsequently recognized. A command phraseis a multiple word command that is recognizable by the voice recognitionengine 205. The voice recognition engine 205 may be implemented asexecutable code stored in memory 117, and that is executed by theapplication processor 101. When the application processor 101 is placedinto a sleep mode, the full functionality of the voice recognitionengine 205 is not available and the speech segment monitor 207 takesover and provides certain features based on detection of speech commandphrase segments. That is, the speech segment monitor 207 provides forvarious operations of the low-power processing 103, and control of otherhardware, in response to voice, even though the application processor101 is in a sleep mode such that the full features of the voicerecognition engine 205 are not available. The voice recognition engine205 becomes available again only in response to waking the applicationprocessor 101 which occurs in response to the user speaking a fullcommand phrase for waking the mobile device 200.

One example of such a low power function or feature is the forceddisplay breath as was described briefly above. FIG. 4 is a flowchart ofan example method for performing a low power mode operation, such as aforced display breath, in accordance with an embodiment. That is, theflowchart of FIG. 4 is one possible example implementation of operationblock 307 in the flowchart of FIG. 3. When the mobile device 200 isplaced into a low power mode of operation such that the applicationprocessor 101 is in a sleep mode, the speech segment monitor 207 isactivated. The speech segment monitor 207 then listens for commandsegments that match the segment speech model 213. In some embodiments,the segment speech model 213 may be stored in an on-board memory of thespeech segment monitor 207. In operation block 401, the speech segmentmonitor 207 may detect a speech segment matching the segment speechmodel 213. As discussed above, the segment speech model 213 may begenerated from the trained speech model 211 or may be derived from thebasic speech model 209 and corresponding default command phrases. Inthat case, the segment speech model 213 need not be a trained speechmodel. That is, the speech command phrase segments of the segment speechmodel 213 may be derived from the basic speech model 209 such that theyare not geared to any specific user speech pattern. In operation block403, when the speech segment monitor 207 detects a speech command phrasesegment, it sends a command to the display manager 215 which thencontrols the display hardware 107 to light the display 109 and therebyforce a display breath. The forced display breath temporarily lights thedisplay 109 to show the display data 221. An example of visualindications 901 that may appear on the display 109 are shown in FIG. 9Awhich was described briefly above. After a preset period of time, thedisplay hardware 107 will again turn the display off as shown inoperation block 405 and the method of operation then ends. The displayhardware 107 may be controlled by the display manager 215 in order toturn off the display 109 after the preset period of time. The displaymanager 215 may determine the breath time period, and may send a commandover the one or more connection buses 105 to command the displayhardware 107 to turn the display 109 off after the duration of theforced breath. If the speech segment monitor 207 also detects that anentire speech command phrase has been spoken, such as a “wake-up”command phrase, then the speech segment monitor 207 will also send acommand to the application processor 101, over the one or moreconnection buses 105, to wake the application processor 101 and therebyimplement any subsequent command that may have been included in thecommand phrase.

FIG. 5 is a flowchart of an example method for revealing helpinformation to a user in accordance with an embodiment. In someinstances, the speech segment monitor 207 may detect repeated uses ofthe same command segments within some predetermined or preset timeinterval. This may be due to any number of reasons one of which may bethat the user is speaking too softly or that the audio environment ispreventing the speech segment monitor 207 from accurately evaluating aspoken command phrase. Thus in operation block 501, if the same commandsegment is detected within a preset time interval, the speech segmentmonitor 207 may command the display manager 215 to force a displaybreath after updating the frame buffer 217 using a specific graphicobject retrieved from the display graphics 223. As shown in operationblock 503, after the time interval is expired, the display manager 215may cause a forced display breath and may also provide a user promptwith help information. More particularly, the user prompt is a graphicobject (i.e. a graphic image) that has been predetermined and is storedin the display graphics 223. The display manager 215 overwrites aportion of the frame buffer 217 using the predetermined graphic object.The speech segment monitor 207 sends an identifier for the predeterminedgraphic object to the display manager 215 along with the command, inresponse to detecting multiple occurrences of the same speech commandphrase segment within a predetermined time period (such as, for example,within 10 or 20 seconds). Examples of predetermined graphic objects thatmay be displayed to the user are illustrated in FIG. 9B and FIG. 9C. InFIG. 9B, if the user has not set up any specific voice controlparameters (i.e. has not set up touchless control) and has not generatedthe trained speech model 211, a message box 903 may be displayed thatinforms the user of the menu selections required to set up touchlesscontrol. In one embodiment, the message box 903 is a pre-created graphicobject that is stored as a display graphic in display graphics 223. Thatis, the display manager 215 may retrieve the message box 903 graphicobject data from display graphics 223, and update a section of the framebuffer 213 in order to revise the display data 221 such that the forcedbreath will include the message box 903 on the display 109. In anotherexample, if the user repeatedly speaks a segment of an existing commandphrase, a message box 905 may be displayed that asks the user whetherthey meant to speak the full command phrase or not and that provides thefull command phrase text to the user. In other words, the message box905 may serve as a reminder of what the full command phrase is in theevent that the user had forgotten. The message box 905 may also be usedto request that the user speak louder or provide some other helpfulinformation. The message box 905 may also be a display graphic objectstored in display graphics 223 and that is used to update the framebuffer 217. As shown in FIG. 9B and FIG. 9C, the message box 903 ormessage box 905 will be displayed along with the visual indications 901during the forced breath. In operation block 505, the display is turnedoff once again and the method of operation ends.

FIG. 6 is a flowchart of an example method for performing a forcedbreath operation on a display in accordance with an embodiment. Themethod of FIG. 6 begins when the mobile device 200, and therefore theapplication processor 101, is operating in a low-power mode as shown inoperation block 601. The display 109 is therefore turned off as shown inoperation block 603. The speech segment monitor 207 then waits for voiceinput in operation block 605. If a speech command phrase segment isdetected in decision block 607, then the speech segment monitor 207 willcommand the display manager 215 to proceed to force a display breath inoperation block 613. Security settings (or other settings) may preventthe forced breath from occurring and therefore these settings arechecked as shown in decision block 611. If the security settings orother settings do not allow for forced breathing to occur, the method ofoperation then ends. The security settings may be stored in the memory117 in some embodiments. If no speech command phrase segment is detectedin decision block 607, then the speech segment monitor 207 continues towait for voice input as shown in operation block 605.

In addition to listening for command phrase segments in decision block607, the speech segment monitor 207 also listens for a complete wake-upcommand phrase as shown in decision block 609. If a wake-up commandphrase is detected, then the speech segment monitor 207 sends a wake-upcommand to the application processor 101 and also passes any speechcommands detected after the wake-up phrase to the application processor101. The application processor 101 then handles any other spokencommands that were subsequent to the wake-up command phrase. If acomplete command phrase was not detected in decision block 609, but acommand phrase segment was detected in decision block 607, then thespeech segment monitor 207 commands the display manager 215 to proceedto force the display breath in operation block 613 in response todetection of the command phrase segment in decision block 607. Thedisplay is again turned off in operation block 615, the applicationprocessor 101 remains in a sleep state, and the method of operationends. However if a complete command phrase was also detected in decisionblock 609, then the speech segment monitor 207 will send a command tothe application processor 101 to wake the application processor 101 asshown in operation block 617. The mobile device 200 therefore exitslow-power mode as shown in operation block 619 and the method ofoperation then ends.

Thus for example, if the display is off and the user begins to speak atrigger phrase such as, “Okay wake up”, the speech segment monitor 207will command the display manager 215 to immediately force a breath assoon as the “Okay” command phrase segment is detected. Among otheradvantages, the mobile device 200 will thereby appear extremelyresponsive by lighting up and showing a “sign of life” even as thetrigger phrase is spoken, and before the mobile device 200 has evendecided that it needs to wake up the application processor 101 fully. Asmentioned above, the “Okay” command phrase segment is not part of thetrained trigger (i.e. it is not part of the complete trained commandphrase) and may not be part of the trained speech model 213.

The method of operation of FIG. 6 may thus be considered as detecting a“compound trigger” such as for example, “Okay wake up” where the “Okay”segment wakes up the display 109 (i.e. results in a forced displaybreath), and the complete trigger wakes up the application processor101. Thus if the display 109 is off and the user speaks “Okay” (in thisexample), the speech segment monitor 207 and display manager 215 willforce a breath to occur, even if the “Okay” command phrase segment isnot followed by the rest of the phrase or a command. The user cantherefore force a display breath at any time, as long as they are inlistening range, simply by saying the command phrase segment “Okay” (orsome other command phrase segment in other examples). If the display 109is already on, then any such command phrase segment will be ignored.Because the command phrase segment utilized (such as “Okay” in theexample above) may not be part of the trained speech model 211, any user(not just the primary user that has gone through the setup process fortouchless control and created a trained speech model 211) will be ableto speak the command phrase segment, such as “Okay”, and force a displaybreath to occur, provided that the current mobile device 200 securitysettings or other settings allow for breathing to occur at that time.

FIG. 7 is a flowchart of an example method for displaying information toa user in accordance with an embodiment. The application processor 101is in a sleep mode of operation as shown in operation block 701 and thedisplay 109 is turned off as shown in operation block 703. The displaymanager 215 waits for a forced breath command in operation block 705. Ifa forced breath is triggered as shown in decision block 707, then thelow-power processing 103 may check user settings to determine whetherthe user has set up voice control as shown in decision block 709. If aforced breath trigger is not detected in decision block 707, then thedisplay manager 215 continues to wait for a forced breath command tooccur in operation block 705.

If voice control has been set up by the user in decision block 709 then,in operation block 719, the display manager 215 may display a reminderof the full command phrase during the forced breath that occurs inoperation block 717, or may retrieve a graphic object showing some otheravailable command phrase or command phrase segments that the user mayemploy. One example of information that may be displayed is shown FIG.9C as message box 905. The display 109 will then be turned off again asshown in operation block 715 and the method of operation ends.

If voice control has not been set up in decision block 709, the displaymanager 215 will force a display breath as shown in operation block 711.The display manager 215 will retrieve the graphic object image data formessage box 903, or a similar message box, from the display graphics 223and overwrite a section of the frame buffer 217 to display it on thedisplay 109 as shown in operation block 713. An example of what may beshown on a mobile device display during execution of operation block 711and operation block 713 is shown in FIG. 9B. That is, the message box903 may provide a prompt to the user informing the user of theappropriate menu selections required to set up voice control. After thebreathing time interval has expired the display will be turned off onceagain in operation block 715 and the method of operation then ends.

FIG. 8 is a flowchart of an example method for prompting a user for avoice command in accordance with an embodiment. The various embodimentsenable provision of user “help” text (as a pre-created graphic object)that may be used to instruct a user to speak louder, remind the user ofwhat their full trigger phrase is, or expose additional one-shotcommands. The application processor 101 is in a sleep mode of operationas shown in operation block 801 and the display 109 is turned off asshown in operation block 803. The speech segment monitor 207 waits forvoice input in operation block 805. If a command phrase segment isdetected in decision block 807, then the display manager 215 will forcea display breath in operation block 809 and will turn the display offagain in operation block 811. The method of operation will then end. Thespeech segment monitor 207 continues to wait for voice input inoperation block 805 until a command phrase is detected in decision block807.

In addition to forcing the display breath in operation block 809, upondetection of a command phrase segment in decision block 807, the speechsegment monitor 207 will also begin a counter and timer operationassociated with the detected command phrase segment. This is illustratedin operation block 813 and continues until a preset number of identicalcommand phrase segments are detected in decision block 815. If the samecommand phrase segment is detected in decision block 815, the counter isincremented as shown in operation block 817. The counter is incrementeduntil the counter reaches a predetermined maximum count in decisionblock 819. If the counter reaches a predetermined maximum count (such asfor example three attempts), then the speech segment monitor 207determines whether the timer has timed out in decision block 821. If thetimer has timed out in decision block 821, then the speech segmentmonitor 207 will continue to wait for voice input in operation block805.

However if the timer has not timed out in decision block 821, such thatthe user has attempted a predetermined maximum amount of command phrasesegment attempts within the preset time interval (for example 3 attemptswithin 20 seconds), then the speech segment monitor 207 will command thedisplay manager 215 to force a display breath in operation block 823.The display manager 215 will retrieve an appropriate display graphicobject from the display graphics 223 and update the frame buffer 217accordingly by overwriting a section. The retrieved display graphicobject will revise the display data 221 to display a prompt to “speaklouder”, to display an icon, or to remind the user of the full commandphrase associated with the command phrase segment. This operation occursin operation block 825, an example of which is illustrated by messagebox 905 in FIG. 9C. The display manager 215 will then turn the displayoff after the forced breath time interval as shown in operation block811 and the method of operation then ends.

Any appropriate information may be displayed to the user in accordancewith the method of operation of FIG. 8. For example, various icons 907may be displayed to the user as shown in the example of FIG. 9D. Theseicons 907 may also be stored in the display graphics 223 and used by thedisplay manager 215 to update the frame buffer 217 and thereby revisethe display data 221 for display during the forced breath operation.

Therefore, among other advantages, the utilization of command phrasesegments contained within the segment speech model 213 enablefunctionality and features that may be accessed during the low-powermode of operation of the mobile device 200, and to reveal these featuresto the user in a progressive manner based on detection of the variouscommand phrase segments or by detection of repeated usage of the samecommand phrase segments within predetermined time intervals. The mobiledevice 200 may include various settings such that a command phrasesegment (user defined or default) that is used to force a breath on thedisplay 109 may either be ON or OFF by default and may be changed by theuser. Although the examples provided above were related to forcing abreath on the display 109 during a low-power operating mode, otherfeatures of functions may also be implemented using the inventiveconcepts herein described. In other words, any function that may beperformed while a device is in a low-power operating mode may benefitfrom the embodiments herein described as may occur to those skilled inthe art. Any of various control systems may benefit from the featuresand functions described herein. One example application is in roboticcontrol systems, where sleep mode is used to conserve power. Voicecontrol segments may be used in low power mode to implement functionswithout waking the main processor of the control system. Thus, smallservo-motors may be operated in low-power mode to move or repositionrobotic equipment slightly without waking and invoking the fullprocessing power of the robotic control system. Other applications willbecome apparent to those of ordinary skill in light of the examples anddescription provided herein. FIG. 10 provides an example apparatus whichis a robotic control system 1000 using the features of voice controldescribed herein. The speech segment monitor 207 is operative to providecommands to control system manager 1215 and further to control system1007 which is operatively coupled to controllable equipment 1009 by acontrol bus 1008. The speech segment monitor 207 is operative to performvarious low-power operations on the equipment 1009 while the applicationprocessor 101 is in sleep mode. For example, in some embodiments, theequipment 1009 may include robotic machinery that may be moved slightlyusing low-power servo motors in response to speech command segmentsdetected by the speech segment monitor 207. The control system manager1215 sends appropriate signals over the one or more connection buses 105to the control system 1007, to operate the equipment 1009, in responseto various speech segments detected by the speech segment monitor 207that match the segment speech model 213. In some embodiments, a controlsystem manager may be implemented by executing control system manager1217 executable code stored in memory 117, where at least one processorof low-power processing 103 is operative to execute the executable codeto perform the features and operations of the control system manager.

While various embodiments have been illustrated and described, it is tobe understood that the invention is not so limited. Numerousmodifications, changes, variations, substitutions and equivalents willoccur to those skilled in the art without departing from the scope ofthe present invention as defined by the appended claims.

What is claimed is:
 1. A computer-implemented method comprising: duringoperation of a mobile device in a low-power mode in which functionalityof a voice recognition engine of the mobile device is reduced:receiving, by data processing hardware of the mobile device, anutterance; while a display of the mobile device is off, detecting, bythe data processing hardware, an initial portion of a predeterminedcommand phrase in the utterance by matching the initial portion of thepredetermined command phrase to a segment speech model associated withthe predetermined command phrase, the predetermined command phrasecomprising a series of multiple words configured to cause the mobiledevice to wake-up from the low-power mode in response to detecting eachword of the series of multiple words in the utterance, and the segmentspeech model storing each word of the series of multiple words of thepredetermined command phrase; and in response to detecting the initialportion of the predetermined command phrase in the utterance: turningon, by the data processing hardware, the display of the mobile device;and displaying, by the data processing hardware for output on thedisplay, at least a portion of a graphical user interface that wasstored in a frame buffer of the mobile device before receiving theutterance.
 2. The method of claim 1, wherein the segment speech model isderived from an untrained speech model that contains a default commandphrase comprising the predetermined command phrase.
 3. The method ofclaim 2, wherein the segment speech model is derived from a trainedspeech model created by a user of the mobile device and containing auser-created command phrase spoken by the user of the mobile device, theuser-created command phrase comprising the predetermined command phrase.4. The method of claim 1, further comprising, in response to detectingthe initial portion of the predetermined command phrase in theutterance: determining, by the data processing hardware, whether aremaining portion of the predetermined command phrase is detected in theutterance within a first predetermined period of time after detectingthe initial portion of the predetermined command phrase in theutterance; and when the remaining portion of the predetermined commandphrase is not detected in the utterance within the first predeterminedperiod of time, automatically deactivating, by the data processinghardware, the display of the mobile device a second predetermined periodof time after displaying the at least the portion of the graphical userinterface for output on the display.
 5. The method of claim 4, furthercomprising, when the remaining portion of the predetermined commandphrase is detected in the second utterance within the secondpredetermined period of time, commanding, by the data processinghardware, the mobile device to wake-up from the low-power mode andoperate in an awake mode.
 6. The method of claim 5, further comprising,during operation of the mobile device in the awake mode in which fullfunctionality of the speech recognition engine is provided, detecting,by the data processing hardware, a voice command in the second utterancethat follows the predetermined command phrase.
 7. The method of claim 6,further comprising, after detecting the voice command in the secondutterance that follows the predetermined command phrase, performing, bythe data processing hardware, an action that corresponds to the voicecommand.
 8. The method of claim 1, wherein the segment speech model isstored in an on-board memory of the mobile device.
 9. A systemcomprising: one or more computers and one or more storage devicesstoring instructions that are operable, when executed by the one or morecomputers, to cause the one or more computers to perform operationscomprising: during operation of a mobile device in a low-power mode inwhich functionality of a voice recognition engine of the mobile deviceis reduced: receiving an utterance; while a display of the mobile deviceis off, detecting an initial portion of a predetermined command phrasein the utterance by matching the initial portion of the predeterminedcommand phrase to a segment speech model associated with thepredetermined command phrase, the predetermined command phrasecomprising a series of multiple words configured to cause the mobiledevice to wake-up from the low-power mode in response to detecting eachword of the series of multiple words in the utterance, and the segmentspeech model storing each word of the series of multiple words of thepredetermined command phrase; and in response to detecting the initialportion of the predetermined command phrase in the utterance: turning onthe display of the mobile device; and displaying, for output on thedisplay, at least a portion of a graphical user interface that wasstored in a frame buffer of the mobile device before receiving theutterance.
 10. The system of claim 9, wherein the segment speech modelis derived from an untrained speech model that contains a defaultcommand phrase comprising the predetermined command phrase.
 11. Thesystem of claim 10, wherein the segment speech model is derived from atrained speech model created by a user of the mobile device andcontaining a user-created command phrase spoken by the user of themobile device, the user-created command phrase comprising thepredetermined command phrase.
 12. The system of claim 9, wherein theoperations further comprise, in response to detecting the initialportion of the predetermined command phrase in the utterance:determining whether a remaining portion of the predetermined commandphrase is detected in the utterance within a first predetermined periodof time after detecting the initial portion of the predetermined commandphrase in the utterance; and when the remaining portion of thepredetermined command phrase is not detected in the utterance within thefirst predetermined period of time, automatically deactivating thedisplay of the mobile device a second predetermined period of time afterdisplaying the at least the portion of the graphical user interface foroutput on the display.
 13. The system of claim 12, wherein theoperations further comprise, when the remaining portion of thepredetermined command phrase is detected in the second utterance withinthe second predetermined period of time, commanding the mobile device towake-up from the low-power mode and operate in an awake mode.
 14. Thesystem of claim 13, wherein the operations further comprise, duringoperation of the mobile device in the awake mode in which fullfunctionality of the speech recognition engine is provided, detecting avoice command in the second utterance that follows the predeterminedcommand phrase.
 15. The system of claim 14, wherein the operationsfurther comprise, after detecting the voice command in the secondutterance that follows the predetermined command phrase, performing anaction that corresponds to the voice command.
 16. The system of claim 9,wherein the segment speech model is stored in an on-board memory of themobile device.
 17. A non-transitory computer-readable medium storingsoftware comprising instructions executable by one or more computerswhich, upon such execution, cause the one or more computers to performoperations comprising: during operation of a mobile device in alow-power mode in which functionality of a voice recognition engine ofthe mobile device is reduced: receiving an utterance; while a display ofthe mobile device is off, detecting an initial portion of apredetermined command phrase in the utterance by matching the initialportion of the predetermined command phrase to a segment speech modelassociated with the predetermined command phrase, the predeterminedcommand phrase comprising a series of multiple words configured to causethe mobile device to wake-up from the low-power mode in response todetecting each word of the series of multiple words in the utterance,and the segment speech model storing each word of the series of multiplewords of the predetermined command phrase; and in response to detectingthe initial portion of the predetermined command phrase in theutterance: turning on the display of the mobile device; and displaying,for output on the display, at least a portion of a graphical userinterface that was stored in a frame buffer of the mobile device beforereceiving the utterance.
 18. The non-transitory computer-readable mediumof claim 17, wherein the segment speech model is derived from anuntrained speech model that contains a default command phrase comprisingthe predetermined command phrase.
 19. The non-transitorycomputer-readable medium of claim 17, wherein the operations furthercomprise, in response to detecting the initial portion of thepredetermined command phrase in the utterance: determining whether aremaining portion of the predetermined command phrase is detected in theutterance within a first predetermined period of time after detectingthe initial portion of the predetermined command phrase in theutterance; and when the remaining portion of the predetermined commandphrase is not detected in the utterance within the first predeterminedperiod of time, automatically deactivating the display of the mobiledevice a second predetermined period of time after displaying the atleast the portion of the graphical user interface for output on thedisplay.
 20. The non-transitory computer-readable medium of claim 17,wherein the operations further comprise when the remaining portion ofthe predetermined command phrase is detected in the second utterancewithin the second predetermined period of time, commanding the mobiledevice to wake-up from the low-power mode and operate in an awake mode.